AUV Linux - Driver Port and Validation

Approach

Install desktop linux development environment
Setup build environment, dependencies, etc. for auv-linux module, check out auv-linux and auv-shared from CVS, download the 3rd party packages.
Get familiar with the software stack and the driver test code by working with a driver for a sensor on a workstation
Setup for 'native' development on the embedded PC104 Lippert board which is connected to the instruments on the auv
Finish porting and validate each driver is correctly receiving and parsing data by running driver test executables (auv-linux/onboard/devices)
Run memory leak and access violation tests using Duma and Electric Fence
Optional - try to get a run of the static analysis tool Coverity
Measure CPU load and power consumption

Status

Executive Summary:

50 days allocated, approx. 2-3 days remaining.
Debian 4.0 Linux has been installed and development environment has been setup and documented for reproducability.
After some bug fixing, Navigation drivers are functioning however a serial port 'byte drop' problem is preventing long runs and is exposing problems in the error handling/recovery logic
Science drivers (Hydroscat, Seabird and Lisst) have been debugged and run via serial port on desktop machines but not exercised thru the terminal servers on AUV.
auv-linux is not ready for deployment, task list detailed below.

Tasks Remaining:

Linux infrastructure (must=1, should=2, nice=3) [1-2 weeks]

[2] Rerun Debian install procedure 'native' on the Lippert board - let the Debian Installer probe hardware and configure linux.
[2] config out un-needed support (audio, etc.)
[2] Boot-time startup of CORBA daemons: omniNames & and notifd -c $OMNINOTIFY_DIR/channel.cfg &
[2] Fix the timing offset issue found in tailcone test (off by 4msec) due to timing diff between 2.6.18 and 2.6.22.
- An option is to update to newer kernel (2.6.24 recommended since it's slated for next debian release 'lenny').

Science Drivers [1 - 2 weeks]

Validation on LanTronics terminal server in Auv

System [1 week]

Run memory leak and access violation tests using Duma and Electric Fence
Measure CPU load and power consumption
Optional - try to get a run of the static analysis tool Coverity
File bugs on issues found

Development Environment [ 1.5 weeks]

Create a Build and Simulation server with the same software versions as auv on a workstation (Test what you fly, fly what you test)
Setup an accessible regression test environment for serial drivers - PC104 + Xtreme104 serial card + device emulators

Skill Requirements for Tasks:

C++, Linux, Termios, Familiarity with Navigation and Science Sensors and Debian Linux

Details:

Install Desktop Linux Development Environment
- Acquired 2Ghz HP x2100 workstation from IT
- Red Hat Enterprise 5 was installed for initial development. This has been re-setup to be Debian 4.0 to match the Linux chosen for the AUV - "Test what you fly, fly what you test".
  - Note: auv-linux developers are typically using Fedora Core 6 (no longer available for download) and have various kernels (2.6.18, 2.6.20 and 2.6.22) since some have run the Fedora Upgrade.
- Recommendation is to setup a build (and simulation execution) machine that has the same linux distribution as the auv
Setup Build Environment
- Worked thru the dependencies required for each 3rd party package.
- Created a new CVS module, auv-pkgs, to archive the versions of 3rd party packages to insure reproducibility.
- Created scripts for automating the install procedure.
Familiarization with software stack and Driver debug/validation on workstation
- Acquired a Hydroscat
- Attached Hydroscat to serial port of workstation
- Found and fixed a couple of minor bugs in the driver
- Ran soak (6 hour) test
Setup for 'native' development on the embedded PC104 Lippert board - the test / build environment for validation and test of the drivers
- Debian 3.1 was found to be incompatible with the auv-linux software and 3rd party packages. The decision was made to update to Debian 4.0
- Since the Lippert Coolrunner II is relatively slow, the installation and setup is painfully time consuming. The approach taken is to perform the installation and setup as mvc-debian4 (dhcp) on the pentium 4 2Ghz desktop and then transfer the harddisk to the PC104 stack and then manually fix up linux for the differences (static IP, mvc-debian, changed ethernet MAC address, etc.)
  - Acquired a 5GB harddisk from IT, purchased a 3.5" to 2.5" adapter for interfacing the notebook driver to the workstation.
  - On the HP workstation, installed Standard distribution of Debian 4.0, worked thru (and documented) dependent packages that need to be added to the standard distribution using Debian package manager (Aptitude / apt-get)
  - Ran into limitation on disk size - purchased 160GB Hitachi drivers (qty 2)
  - After a successful auv-linux build and test of Hydroscat driver, moved the harddrive to the CTD AUV Vehicle (DMO1).
- Fixed up the ethernet and configured the linux kernel for the serial ports for the AUV.
  - Configured and rebuilt the linux kernel for the Xtreme104 serial card
- Created Instructions for Auv Debian 4.0 Linux Install and configuration (re-used by FOCE team). AUV Debian4 Linux and auv-pkgs Install, build
- TODO (must=1, should=2, nice=3)
  - [2] Rerun Debian install procedure 'native' on the Lippert board - let the Debian Installer probe hardware and configure linux.
  - [1] Install Fan Card driver
  - [1] Configure eth1 for wifi
  - [2] config out the un-needed audio support
  - [1] rebuild the kernel to add the two serial ports not setup at boottime (ttyS12 adn ttyS13)
  - [2] Boot-time startup of CORBA daemons: omniNames & and notifd -c $OMNINOTIFY_DIR/channel.cfg &
  - [2] Fix the timing offset issue found in tailcone test (off by 4msec) due to timing diff between 2.6.18 and 2.6.22.
    - An option is to update to newer kernel (2.6.24 recommended since it's slated for next debian release 'lenny').

Finish porting and validate each driver is correctly receiving and parsing data by running driver test executables (auv-linux/onboard/devices)
- Serial Device driver.
  - Verify serial port throughput capabilities (using SerTest test program) and resolve linux kernel driver, fifo depth, or IRQ assignments to fix byte drop.
    - Write separate serial test program (SerTest) to validate linux kernel driver and hardware configuration (IRQ)
- Navigation Drivers (first priority)
  - ashtec(gps) - 'Partial' Basic functional - tested with no statellite acquisition. Seg faults with ctrl-c (exit).
  - bluefinBatt - Established serial connection. Character echo is interfering with the driver, action for Hans to modify jumper on Xtreme104 for 'no echo' on RS485.
  - 3dmgx1 Microstrain AHRS Compass - Tom O ported the 3dmgx1 driver. The is passing serial data though there are issues with the data stability (could be the compass).
    - crossbow AHRS compass - Hans requested focus on this driver first (July 2008), however it will likely not be on a deployed vehicle since 3dmgx1 is preferred.
  - parosci Paroscientific pressure sensor - driver is functional however failing overnight tests. (Last minute update: The latest pull from CVS failed)
  - tailcone - tested and compared to QNX. Found a timing offset issue that can be fixed by upgrading the linux kernelt to 2.6.22 or newer (2.6.24 recommended which should be available in Debian 'Lenny' in 2H08 (Sept. best guess).
  - teledyne DVL - driver is functional however failing overnight tests.
- Science Drivers (second priority)
  - Seabird
    - In progress... getting an error with seabird25 - 06:16:34.747 ERROR [AttributeParser] Attributes::parse() - Couldn't open CTD attributes file "tcp:sci1-dmo1:10002"
  - Hydroscat - HS2 Backscatter optical sensor
    - Actions
      - Fixed bug in sscanf on snorm values (%hx -> %1hx). Added 'STOP' to init. Fixed segfault in log write. Fixed segfault when run with terminal server.
  - Biolume - Biolumenescence
    - Status - Not tested yet
  - lisst - Laser based particle size analyzer
    - Added 'STOP' to init. debugged and running from Tom's FC6 workstation 6 Mar 08
  - Gulper
    - no linux driver currently available (should be a simple 'fire' command)
Run memory leak and access violation tests using Duma and Electric Fence
Optional - try to get a run of the static analysis tool Coverity
Measure CPU load and power consumption

Recommendations:

Linux build: Setup a build (and simulation execution) machine that has the same Debian 4.0 linux distribution as the auv
Regression testing: Setup a PC104 stack with Xtreme104 serial card and attach device emulators (microprocessors) on the serial ports for regression testing (Fly what you test, test what you fly).
- Device emulators for each class (Streaming, Polled, Fixed, Variable).
- Inject errors for code coverage on error handlers.

Thom's Notes:

Time on AUV was 1999.

Fresh checkout from cvs, build failed.
gcc -o uplinkQserver uplinkQueuerServer.o /home/auv/prj/auv-linux/onboard//utils/auvUtils.a /home/auv/prj/auv-shared/lib/libAuvShared.a -L/opt/omniOrb/omniORB-4.1.0/build/lib/ -L/usr/local/apr/lib -lgctpc -lpthread -llog4cxx -lnetcdf -lnewmat -lapr-1 -laprutil-1 -lrt
uplinkQueuerServer.o: In function `main':
/home/auv/prj/auv-linux/onboard/tools/uplinkQueuerServer.cc:9: undefined reference to `mbari::Logger::configure(char const*)'
/home/auv/prj/auv-linux/onboard/tools/uplinkQueuerServer.cc:17: undefined reference to `MsgQueuerServer::MsgQueuerServer(char const*, long, long, unsigned char, int)'
/home/auv/prj/auv-linux/onboard/tools/uplinkQueuerServer.cc:26: undefined reference to `MsgQueuerServer::run()'
collect2: ld returned 1 exit status
make[1]: *** [uplinkQserver] Error 1
make[1]: Leaving directory `/home/auv/prj/auv-linux/onboard/tools'
make failed
make: *** [all] Error 2

The build got far enough to enable teledyne and crossbow builds. Teledyne would not run without creating 'latest' directory in onboard/logs though I seem to remember there was a link involved (talk to Tom O.).

For Leak detection:

http://www.cprogramming.com/debugging/valgrind.html

http://www.tldp.org/HOWTO/Valgrind-HOWTO/\\

Attachments:

interfaces (text/plain)

serial.conf (application/octet-stream)

teledyne.test.results (text/plain)

teledynedvl.testresults.1Apr08 (text/plain)

parosci.testresultssoak.2Apr08 (text/plain)

Document generated by Confluence on Feb 04, 2026 08:05